I tested the demo app on iOS and the results vere pretty mid, at least for the S version. Not sure if this is the nature of the model, but just saying

Probably the nature of the model. You can share what results you get and I can do a cross-check.

I tested the demo app on iOS and the results vere pretty mid, at least for the S version. Not sure if this is the nature of the model, but just saying

From what I've tested, the S variant is fine for simple segmentation when objects don't overlap, but for more complex scenes it's true that artifacts show up. The X variant however worked fine on all images I tried, even ones with quite complex scenes. Did you observe bad performance on X variant also?

I tested the demo app on iOS and the results vere pretty mid, at least for the S version. Not sure if this is the nature of the model, but just saying

From what I've tested, the S variant is fine for simple segmentation when objects don't overlap, but for more complex scenes it's true that artifacts show up. The X variant however worked fine on all images I tried, even ones with quite complex scenes. Did you observe bad performance on X variant also?

I gave some more testing to the X version, and the point/box based detections look cool, but the prompt-based ones were pretty bad. I almost never got the result I expected, but maybe its due to poor quality of text embeddings. quick example:

@msluszniak @chmjkb I've:

• updated docs by adding the section on selector use to useInstanceSegmentation.md, the previous file didn't really fit under hooks/ in my opinion.
• moved the helper function bboxArea to utils/commonVision.ts as requested.
• fixed keyboard handling - hopefully it feels better now.
• fixed a problem with the way cropped images were passed to CLIP for embeddings - previously I followed exactly how it is done in ultralytics Python implementation where they simply pass the cropped image without masking; now the cropped image is masked based on the segmentation mask, so it should work better on examples as the one posted above. There are still some problems with text prompts on some images, but these are because of CLIP model not being able to properly embed images with certain parts of the image masked.
• added optional topk parameter to selectByText that enables the user to return top-k best matching instances to the given text prompt. This is mostly for convenience and it doesn't truly solve the problem of returning multiple instances as topk must be passed explicitly, so e.g. it doesn't cover use cases where user would like to automatically count the number of objects matching certain prompt. The problem is that text/image embeddings are inherently contrastive and passing like a threshold doesn't really solve it. I could add a function that does something like an open-vocabulary classification where the user inputs text prompts that correspond to classes and for each segmented instance we return the best matching class based on cosine similarity, but I'm not sure if that's needed. What do you think?

I will be also adding benchmarks shortly.

• added optional topk parameter to selectByText that enables the user to return top-k best matching instances to the given text prompt. This is mostly for convenience and it doesn't truly solve the problem of returning multiple instances as topk must be passed explicitly, so e.g. it doesn't cover use cases where user would like to automatically count the number of objects matching certain prompt. The problem is that text/image embeddings are inherently contrastive and passing like a threshold doesn't really solve it. I could add a function that does something like an open-vocabulary classification where the user inputs text prompts that correspond to classes and for each segmented instance we return the best matching class based on cosine similarity, but I'm not sure if that's needed. What do you think?

@msluszniak @chmjkb I've:

• updated docs by adding the section on selector use to useInstanceSegmentation.md, the previous file didn't really fit under hooks/ in my opinion.
• moved the helper function bboxArea to utils/commonVision.ts as requested.
• fixed keyboard handling - hopefully it feels better now.
• fixed a problem with the way cropped images were passed to CLIP for embeddings - previously I followed exactly how it is done in ultralytics Python implementation where they simply pass the cropped image without masking; now the cropped image is masked based on the segmentation mask, so it should work better on examples as the one posted above. There are still some problems with text prompts on some images, but these are because of CLIP model not being able to properly embed images with certain parts of the image masked.
• added optional topk parameter to selectByText that enables the user to return top-k best matching instances to the given text prompt. This is mostly for convenience and it doesn't truly solve the problem of returning multiple instances as topk must be passed explicitly, so e.g. it doesn't cover use cases where user would like to automatically count the number of objects matching certain prompt. The problem is that text/image embeddings are inherently contrastive and passing like a threshold doesn't really solve it. I could add a function that does something like an open-vocabulary classification where the user inputs text prompts that correspond to classes and for each segmented instance we return the best matching class based on cosine similarity, but I'm not sure if that's needed. What do you think?

I will be also adding benchmarks shortly.

I think the topk should be enough, no need to over-engineer this. I'll review it now and if it looks ok then